With the growing popularity of Internet video and Internet TV, there is an increasing need for streaming solutions that allow a synchronized user experience for users between different types of streams received by one or more devices. For example, a Hybrid Broadcast Broadband TV (HBBTV) media processing device may be capable of receiving and processing a primary stream, e.g. a live broadcast, in parallel with one or more secondary streams wherein the secondary streams may convey additional information, such as an additional audio stream, a foreign language subtitle stream, or an interactive “event” stream, to provide added value to the primary TV signal.
Furthermore, in so-called second screen or companion screen applications, the primary stream may be rendered by a first media processing device, e.g. a TV or the like, and the content of the secondary stream may be rendered by a second media processing device, e.g. an electronic table or an audio system associated with the TV. When processing a primary and secondary stream in the above-mentioned scenarios, a temporal relation between the primary and secondary stream needs to be maintained in order enable the media processing device to render these streams in sync to the user. Techniques for synchronization of one or more streams by a single media processing device is usually referred to as inter-stream synchronization which should be distinguished from other known synchronization techniques such as intra-stream synchronization and inter-destination media synchronization.
Certain problems may occur implementing such scenarios. One problem is associated with the fact that often the primary and secondary stream are being transmitted as independent streams to the media processing device(s) on the basis of different access technologies (e.g. the secondary stream is delivered on the basis of MPEG DASH as a third-party over-the-top (OTT) service, while the primary stream is delivered as a DVB-type broadcast service). Hence, data units of the primary and secondary stream may have different data formats, originate from different media sources and experience different network delays during transmission.
A further problem may arise in cases where the secondary stream is lagging behind in time with respect to primary stream. For example, in case the primary stream is a live stream, often the secondary stream can only be created after the content of the primary stream is made available by broadcasting. For example, if the secondary stream is created by a third-party, the third party content creator will receive the primary stream at approximately the same time as the media processing devices. An example of such secondary stream is a video stream of a sign language interpreter for the deaf or hearing impaired, foreign language subtitles created on the fly, or third party commentary tracks for a football match, or foreign language audio translations of a live news feed. Creating such secondary stream will take a few seconds by the third party content creator, after reception of the primary stream.
In that case, the secondary content is only available many seconds after the corresponding primary content has been received, so that in order to realize synchronization between the primary and the secondary stream, the primary stream has to be delayed by the media processing device for a relatively large amount of time. Hence, once a user activates the secondary stream, or switches to a new primary stream, the screen will freeze or rewind for a couple of seconds to allow the secondary stream to catch up, thereby seriously degrading the user experience. This problem may be referred to as the instant inter-stream synchronization problem.
Typically synchronization is realized in two steps. First the asynchronicity between receivers is determined, by for example collecting individual status reports from the media receivers (e.g. the DVB and/or IP receivers) on the current timing of the content being received (e.g. the last timestamp received for a particular media stream). Thereafter, in a second step, the synchronicity is corrected. The source, receiver or an intermediate node adjusts the playout or throughput based on the current and desired playout position. This adjustment can be done by either delaying or advancing the media stream (or local buffer) until the synchronization is restored.
Various known techniques such as watermarking, audio fingerprinting or matching of timestamps may be used to synchronize a secondary stream with a primary stream. For example, Nielsen and Civolution use audio fingerprinting or audio watermarking respectively to synchronize an interactive feed with the live broadcast. However, these techniques do not provide a solution for the instant inter-stream synchronization problem where the interactive content (the secondary stream) is only available seconds after the primary stream.
WO211144775 describes a method for synchronizing real-time subtitling with audio by either creating a new stream in the network on the basis of the primary stream and the secondary stream that carries the subtitles or by temporarily delaying the primary steam in the TV until the secondary stream arrives. Forming a new stream in the network will be difficult to implement if the primary and secondary stream are based on different, non-related transport protocols, e.g. a DVB primary stream and an MPEG-DASH or an RTP secondary stream. Furthermore, it will double the bandwidth that is required to broadcast a television channel to receivers. Delaying the primary stream at the receiver side will require a user to wait thereby considerably degrading the user experience.
Hence, there is a need in the art for improved methods and systems that enable synchronized data processing a primary stream, in particular a primary broadcast stream, and a secondary stream by a receiver wherein the secondary stream becomes available to the receiver only many seconds after the primary stream is available.